Some technical details on confidence intervals for LIFT measures in data mining

A LIFT measure, such as the response rate, lift, or the percentage of captured response, is a fundamental measure of effectiveness for a scoring rule obtained from data mining, which is estimated from a set of validation data. The LIFT measures are related to the ROC (Receiver Operator Characteristic), but there exist some important differences. In this paper, we study how to construct confidence intervals of the LIFT measures. We point out the difficulty of this task and explain how simple binomial confidence intervals can have incorrect coverage probabilities, due to omitting variation from the sample percentile of the scoring rule. We derive the asymptotic distribution using some advanced empirical process theory and the functional delta method in ∗Technical Report 14-02, Department of Statistics, Northwestern University. †Wenxin Jiang is Professor of Department of Statistics, Northwestern University, Evanston, IL 60208 (email: [email protected]); and Yu Zhao is Statistician at Amazon (email: [email protected]).